Direct three-dimensional pointing using light tracking and relative position detection

ABSTRACT

A computing system for direct three-dimensional pointing includes at least one computing device, and a pointing/input device including at least one light source and a motion sensor module for determining absolute and relative displacement of the pointing/input device. At least one imaging device is configured for capturing a plurality of image frames each including a view of the light source as the pointing/input device is held and/or moved in a three-dimensional space. Two or more imaging devices may be provided. A computer program product calculates at least a position and/or a motion of the light source in three-dimensional space from the plurality of sequential image frames and from the pointing/input device absolute and relative displacement information, and renders on the graphical user interface a visual indicator corresponding to the calculated position and/or the motion of the light source. Methods for direct three-dimensional pointing and command input are described also.

This utility patent application claims the benefit of priority in U.S.Provisional Patent Application Ser. No. 62/028,335 filed on Jul. 24,2014, the entirety of the disclosure of which is incorporated herein byreference.

TECHNICAL FIELD

The present disclosure relates to human-computer interaction systems.More specifically, this disclosure pertains to methods and systemsrelated to three-dimensional pointing, using a system integrated forboth absolute position detection and relative position detection.Embodiments of a pointing/input device useful in the methods and systemsare disclosed also.

BACKGROUND OF THE INVENTION

As shown in FIG. 1, for three-dimensional (3D) pointing on atwo-dimensional (2D) screen, a user O uses a handheld 3D pointing device2 aimed at a point on a display surface 3 of a 2D image display device4. In systems wherein the pointing device 2 emits a beam of light (seebroken line), the point is the location where the light beam contactsthe display surface. Example pointing devices 2 include a computermouse, a video game console, or a so-called “smart” TV remote. Thedisplay surface 3 may be the monitor screen of a computing device, avideo game console, etc.

In systems where a pointer such as a cursor, cross-hair, or otherindicator is always displayed at the point of the display surface wherethe pointing device 2 is aimed, the pointing device 2 is referred to asa direct or absolute pointing device and the process is referred to as adirect or absolute pointing process. On the other hand, in systems wherethe pointer is not necessarily displayed at the point of the displaysurface where the pointing device 2 is aimed, the pointing device 2 isreferred to as a relative pointing device and the process is referred toas a relative pointing process. These concepts are depicted in moredetail in FIGS. 2 and 3.

FIG. 2A depicts an initial status of a relative pointing process. Anaimed point 20 and a displayed point 21 are set by the system tocoincide. Once the aimed point 20 of the pointing device 2 leaves aboundary of the display surface (see FIG. 2B), the displayed point 21does not follow but rather remains at a point X which represents thespot where the aimed point 20 departed from the display surfaceboundary. In such systems, the accumulated displacement travelled by thepointing device 2 outside the boundary of the display surface isrecorded and added to a final location of the displayed point 21 whenaimed point 20 returns within the boundary of the display surface.Therefore, as shown in FIG. 2C, the displayed point 21 is different fromthe aimed point 20 after this process. This significantly departs fromthe cognition and intuition of an operator. What an operator needs is apointing device that can always do direct pointing, i.e., the displayedpoint 21 always coincides with the aimed point 20, even when thepointing device 2 leaves the display surface 3 and reenters the displaysurface 3 again, as shown in FIG. 3C.

However, almost all modern 3D pointing devices and 3D remotes such asfor smart televisions are relative pointing devices. This is becausesuch devices use a motion sensor module to provide information fordetermining the location of a pointer on the device screen, and theinformation provided by the motion sensors is relative information. Asan example, a motion sensor module may include a G-sensor, a gyroscopesensor, and a magnetic field sensor. These sensors provide rotationinformation (azimuth, pitch, and roll) and acceleration/velocityinformation (linear acceleration, angular velocity) relative to theprevious location of the pointing device 2. A number of compensationtechniques are known to correct a difference between an aimed point 20and a displayed point 21 of a pointing device 2. However, because thelocation of the pointing device 2 relative to the display surface is notknown, such compensation techniques are of reduced effectiveness.

In addition to a motion sensor module, it is known to include an imagingdevice in a pointing device to provide additional information fordetermination of an aimed point. As shown in FIG. 4, an imaging devicesuch as an infrared (IR) camera 32 may be included in the pointingdevice 2 whereby images of light emitted by light sources 34 in a sensorbar 36 are used together with information provided by a motion sensormodule 30 to precisely identify an aimed point 38 of the pointing device2 and to provide a pointer or other icon coinciding with that aimedpoint 38 on a display surface 40. This scheme is used in a number ofdevices, such as Nintendo's® Wii®, Philips'® uWand, and certaintechnology of Apple®.

The system as depicted in FIG. 4 has certain disadvantages. Because theimaging device 32 is installed in the pointing device 2, the operatormust ensure that the pointing device 2 constantly faces the sensor bar36 so that the imaging device can capture images of the sensor bar 36.In turn, because computation to determine a location of aimed point 38is done by modules contained in the pointing device 2, the system cannottake advantage of the full computing power of a console 42. Ifresponsibility for such computation is transferred to console 42, thenthe operation of the pointing device 2 would not be convenient for usersif the connection between the pointing device 2 and console 42 is wired.Alternatively, in a wireless configuration high bandwidth wirelesstransmission is required between the pointing device 2 and the console42. Moreover, the system cannot take advantage of any imaging capabilityof display surface 40 (for example, an integrated or add-on web cam).Still further, improvements in the imaging capability of a system suchas that shown in FIG. 4 are not possible due to space considerations,since typically adding additional imaging devices to pointing device 2is not possible due to space considerations, i.e. because the pointingdevice 2 simply cannot accommodate additional imagers. The only solutionis to attempt to improve resolution of the imaging device included inpointing device 2, which may be undesirable due to cost considerationsand the limitations of resolution of even the most effective imagingdevices sized to fit in or on a pointing device 2 suitable for hand-heldoperation by a user.

There is accordingly a need identified in the art for direct pointingsystems that can utilize the computing capabilities of a console ratherthan relying only on the computing power provided by a pointing device,and also which can utilize imaging devices incorporated into or added-onto a display surface such as a computing device screen, smart TV screen,etc. The ability to interface with multiple imaging devices would alsobe desirable. Also, such systems would provide that a displayed point ofa pointing device on a display surface always coincides with an aimedpoint of the pointing device, even when that aimed point departs from aboundary of a display surface and then reenters that boundary.

SUMMARY OF THE INVENTION

To solve the foregoing problems and address the identified need in theart, the present disclosure provides a direct pointing system that cantake advantage of the computing power of the console and the imagingcapability of a display surface equipped with one or more imagingdevices. The described system can be operated with more flexibility, andcan still provide good direct pointing results.

In one aspect, the present disclosure describes a computing system fordirect three-dimensional pointing and command input. The system includesat least one computing device and a graphical user interface. Apointing/input device is provided including at least one light sourceand a motion sensor module. The motion sensor module providesinformation regarding an absolute and a relative displacement of thepointing/input device to the at least one computing device. At least oneimaging device is operably linked to the computing device processor, andmay be associated with the graphical user interface. In embodiments, aplurality of imaging devices are associated with the graphical userinterface and operably linked to the computing device processor.

The imaging device is configured for capturing a plurality of imageframes each including a view of the at least one light source as thepointing/input device is held and/or moved in a three-dimensional spaceand within a field of view of the at least one imaging device. A onenon-transitory computer program product operable on the computing deviceprocessor includes executable instructions for calculating at least aposition and/or a motion of the at least one light source inthree-dimensional space from the plurality of sequential image framesand from the pointing/input device absolute and relative displacementinformation, and for rendering on the graphical user interface a visualindicator corresponding to the calculated position and/or the motion ofthe at least one light source.

In embodiments, the computer program product includes executableinstructions for determining a position and/or a motion of the at leastone light source in three-dimensional space by determining a currentposition of the at least one light source in a captured image frame anddetermining a prior position of the at least one light source in atleast one prior captured image frame. A relative displacement of thepointing/input device is determined by the computing device processorfrom information provided by the motion sensor module. Next a locationof the at least one light source in three-dimensional space iscalculated from the determined relative displacement information and thedetermined at least one light source current and prior positions. Apointing direction of the pointing/input device defining an axis of thepointing/input device is determined from information provided by themotion sensor module. From the calculated three-dimensional location ofthe at least one light source and the determined pointing direction ofthe pointing/input device provided by the motion sensor module, anintersection point of the axis of the pointing/input device and thegraphical user interface is calculated by the computing device processorand displayed as a visual indicator (an icon, cross-hairs, a pointer,etc.) in the graphical user interface. Movements of the pointing/inputdevice may be interpreted as particular command inputs by the computingdevice processor.

In embodiments, the computer program product includes executableinstructions for calculating a location of the at least one light sourcein three-dimensional space from the determined current and priorpositions of the at least one light source by identifying a first regioncorresponding to a position of the at least one light source in thecaptured image frame and a second region corresponding to a position ofthe at least one light source in the at least one prior captured imageframe. From those identified regions, a first position of the at leastone light source in the captured image frame and a second position ofthe at least one light source in the at least one prior captured imageframe are calculated. Then, a displacement vector of the pointing/inputdevice caused by translating the at least one light source between thefirst region and the second region is calculated. This process isrepeated in succeeding sets of current and prior captured image frames.

In another aspect, a method for tracking a pointing/input device isprovided using the light source position and pointing/input deviceabsolute and relative displacement information obtained from the systemas described above.

These and other embodiments, aspects, advantages, and features will beset forth in the description which follows, and in part will becomeapparent to those of ordinary skill in the art by reference to thefollowing description and referenced drawings or by practice of theinvention. The aspects, advantages, and features are realized andattained by means of the instrumentalities, procedures, and combinationsparticularly pointed out in the appended claims. Unless otherwiseindicated, any patent and/or non-patent citations discussed herein arespecifically incorporated by reference in their entirety into thepresent disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings incorporated in and forming a part of thespecification, illustrate several aspects of the present invention, andtogether with the description serve to explain the principles of theinvention. In the drawings:

FIG. 1 depicts a user using a prior art system for three-dimensionalpointing using a handheld pointer;

FIG. 2 illustrates a prior art system and method for a relative pointingprocess, showing an aimed point and a displayed point coincidinginitially (FIG. 2A), the aimed point leaving the display surface and thedisplayed point remaining (FIG. 2B), and the aimed point returning tothe display surface at a different region from the displayed point (FIG.2C);

FIG. 3 illustrates a prior art system and method for a direct pointingprocess, showing an aimed point and a displayed point coincidinginitially (FIG. 3A), the aimed point leaving the display surface (FIG.3B), and the aimed point returning to the display surface (FIG. 3C);

FIG. 4 shows a prior art system for three-dimensional pointing,including a pointer having an integrated IR camera and a display surfaceassociated with a sensor bar;

FIG. 5 shows a system for three-dimensional pointing and command inputaccording to the present disclosure, including a pointing/input deviceand a display surface including an imaging device;

FIG. 6 depicts a pointing/input device according to the presentdisclosure;

FIG. 7 illustrates computation of a displayed point on a display surfaceaccording to the present disclosure;

FIG. 8 shows in flow chart form computations according to the presentdisclosure for direct three-dimensional pointing using a single imagingdevice associated with a display surface;

FIG. 9 illustrates obtaining image frames over a time sequence of t_(k),t_(k+1), t_(k+2);

FIG. 10 illustrates computation of a displayed point on a displaysurface according to FIG. 7 in greater detail;

FIG. 11 illustrates in isolation computation of a displacement vector ofa pointing/input device shown in FIG. 10;

FIG. 12 illustrates data obtained by a motion sensor module of apointing/input device according to the present disclosure;

FIG. 13 shows a system for three-dimensional pointing and command inputaccording to the present disclosure, including a pointing/input deviceand a display surface including two imaging devices;

FIG. 14 shows in flow chart form computations according to the presentdisclosure for direct three-dimensional pointing using the system ofFIG. 13; and

FIG. 15 illustrates computation of a displayed point on a displaysurface using the system of FIG. 13.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the illustrated embodiments,reference is made to the accompanying drawings that form a part hereof,and in which is shown by way of illustration specific embodiments inwhich the invention may be practiced. These embodiments are described insufficient detail to enable those skilled in the art to practice theinvention. Also, it is to be understood that other embodiments may beutilized and that process, reagent, materials, software, and/or otherchanges may be made without departing from the scope of the presentdisclosure.

At a high level, the present disclosure provides systems and methods forobtaining an absolute location of a visual marker, including a pointersuch as a cursor, cross-hair, etc. on an image display apparatus usingintegrated absolute and relative devices. With reference to FIG. 5, thesystem includes a pointing/input device 50, an image display device 52,at least one image capture device 54, and a computing device 56. Withoutintending any limitation, example image display devices include suchdevices as a projector, a television screen, a graphical user interfacesuch as a computer monitor (which may be 2D or 3D), and the like.Example image capture devices include webcams, etc. example computingdevices include computing devices (personal computers, smartphones,tablets, laptops, etc.), gaming consoles, and the like. It will beappreciated that a transferring protocol is established between theimage capture device 54 and the computing device 56. Such protocols maybe wired or wireless as is known in the art.

In use, the pointing/input device 50 functions substantially in themanner of a laser pointer, except it moves an indicator such as acursor, cross-hairs, etc. instead of a red dot. When an operator 0 usesthe pointing device 50 to aim at a point (e.g., point 70 in FIG. 5) onthe image display apparatus 52, a cursor will appear at the locationpointed to by the pointing device 50. This cursor will move when thepointing device 50 changes its direction, but always to a location onthe image display apparatus 52 at which the pointing device 50 is aimed.

The pointing/input device 50 of this disclosure can also be used as aninput device, substantially similar in function to a computer mouse orother such input device. The position specified by the pointing/inputdevice 50 on the image display apparatus 52 on an x-y coordinate systemdefined on the image display apparatus 52 is computed, and (x, y)coordinates of that specified position can be used to identify an itemor icon displayed on the image display apparatus 52. Therefore, bymanipulating the pointing/input device 50, a user can interact with mostoperating systems (e.g., Android® or Microsoft® Windows®), for exampleselecting files, programs, or actions from lists, groups of icons, etc.,and can freely move files, programs, etc., issue commands or performspecific actions, such as those used in a drawing program.

At least three components are embedded in the pointing/input device 50(FIG. 6): a light source 62, a control panel 64, and a motion sensormodule 66. In an embodiment, the light source 62 is an LED light, butother suitable light sources are contemplated. The control panel 64consists of multiple buttons, which can provide direct functionality,such as the number keys, arrow keys, enter button, power button, etc.The system uses images of the light source 62 taken by the image capturedevice 54 and information provided by the motion sensor module 66 toidentify a location pointed to by the pointing/input device 50 as willbe described below. An absolute position of an icon such as a cursor ora cross-hair, representing an aimed point of the pointing/input device50 on the image display apparatus 52 can then be precisely computed.

The motion sensor module 66 consists of a set of motion detectingsensors to provide absolute and relative motion information of thedevice (e.g., acceleration, rotations, etc) to the computing device 56in real time through some wireless channel. The set of motion detectingsensors contained in the motion sensor module 66 can include a G-sensor,a gyroscope sensor, a magnetic field sensor, and others.

The image capture device 54 functions as a viewing device for thecomputing device 42, taking images of the scene in front of the imagedisplay apparatus 52 at a fixed frame rate per second. Those images aresent to the computing device 56 for subsequent processing. Use of anyconventional image capture device/imaging device 54, including singlelens imaging devices such as a standard webcams, is contemplated. In anembodiment, an image capture device 54 having a frame rate that is atleast 30 frames per second is contemplated.

The computing device 56 processor provides the functionality of lightsource location identification that will identify the location of thelight source 62 in sequential images sent by the image capture device54, and of computation of the point that the pointing/input device 50points to on the image display apparatus 52 (see FIG. 7 for anotherillustration, which is a projection of the scene in the xz-plane). Whenthe computing device 56 receives an image from the image capture device54, it first identifies the location of the light source 62 in the imageusing light source tracking software. It next uses the identifiedlocation of the light source 62 in a previous image sent by the imagecapture device 54, and relative displacement information sent by themotion sensor module 66 of the pointing device 50 to compute thelocation of the light source 62 in 3D space. Finally it uses the 3Dlocation of the light source 62 and orientation angle information sentby the motion sensor module 66 to compute an intersection point 70 of anaxis of the pointing/input device 50 defining an aimed direction of thepointing/input device 50 with the image display apparatus 52 (see FIG.7). This is the point on the image display apparatus 52 that thepointing/input device 50 points to. Therefore, by displaying a cursor orother icon at that point of the image display apparatus 52, the job ofperforming direct pointing is done.

The computation process of the computing device 56 processor is shown inflow chart form in FIG. 8. Here, for ease of explanation, we assume theimaging device 54 is installed at the center of the image displayapparatus 52 with its focal point 72 aligned with the screen of theimage display apparatus 52. However, alternative configurations arecontemplated as will be discussed further below. A right-handedcoordinate system is defined for the scene, with the origin of thecoordinate system set at the center point of the image display apparatus52 which is also the focal point of the image capture device 54 Thex-axis is a horizontal line in the image display apparatus 52 thatpasses through the center point, and the y-axis is a vertical line inthe image display apparatus 52 that passes the center point. Allsubsequent figures are projections of the scene on the xz-plane which isparallel to the floor of the room. In the following discussion, weexplain the computation process in detail.

In Step 80, the computing device 56 processor identifies the regioncorresponding to the light source in the image taken at time t_(k+1)(see FIG. 9) by the image capture device 54. The identification processis carried out using light tracking techniques and associated software.One suitable light tracking technique is disclosed in the presentAssignee's U.S. patent appl. Ser. No. 14/089,881 for “Algorithms,Software, and an Interaction System that Support the Operation of an Onthe Fly Mouse,” the entirety of the disclosure of which is incorporatedherein by reference.

The center of the region will be denoted 74 in the image recordingportion 76 (such as a CCD) of the image capture device 54 (see FIG. 7and FIG. 10). That point 74 is then mapped to a point 102 on a virtualimage plane 1004 of the same dimension as the image display apparatus 52through the focal point 72 of the image capture device 54 (see FIG. 10).At this time we assume the same process has been performed for the imagetaken at time t_(k) (see FIG. 9), that is, the center point of theregion corresponding to the light source in the image taken at timet_(k) was identified, denoted 78, and mapped to a point 1006 in thevirtual image plane 1004 through the focal point 72 (see FIG. 10). Ifthe point 74 is the same as the point 78, we repeat the process for thesubsequent captured frames until we reach a time t_(k+i+1) for some i>0such that the points we obtained for t_(k+i) and t_(k+i+1) aredifferent. For simplicity of notation, these points will also be denoted74 and 78. Note that in FIGS. 10, 74 and 78 represent different pointseven though they seem to coincide at the same point.

In Step 82, the computing device 56 processor computes the location ofthe light source 62 in 3D space using information obtained from Steps 88and 90. Since the operator 0 can move and rotate the pointing device 50,the light source 62 experiences both translation and spinning. Steps 88and 90 provide a displacement vector of the light source 62 betweent_(k) and t_(k+1) (or, t_(k+i) and t_(k+i+1) for some i>0. However, forsimplicity, we shall assume i=0 here) to Step 82. To make illustrationeasier, we enlarge the portion of FIG. 10 that is inside the dottedcircle 1008 and show the enlarged portion in FIG. 11. In FIG. 11, thepoints 1102 and 1104 represent the locations of the light source 62 att_(k) and t_(k+1), respectively, and the point 1106 represents the spinpivot of the pointing device 50 at t_(k+1). The displacement vector ofthe light source 62 between t_(k) and t_(k+1), denoted {right arrow over(V)}1108, is the vector between point 1102 and point 1104. If we know{right arrow over (V)} 1108, then we can compute 1102 and 1104 asfollows: find a point A on the line 1110 defined by the image point 74and the point 1112 on the virtual image plane 1004, and find a point Bon the line 1114 defined by the image point 74 and the point 1116 on thevirtual image plane 1004 (see FIG. 10 or FIG. 11) so that

B−A={right arrow over (V)}

Since the parametric representations of 1110 and 1114 are known, findingA and B is a straightforward process, and 1102 is A and 1104 is B. So,the task here is to find {right arrow over (V)}1108. This process isdiscussed below.

If we use r(t) to represent the location of the light source 62 on animage at time t, then the displacement vector {right arrow over (V)} canbe expressed as

$\overset{\rightarrow}{V} = {{{r\left( t_{k + 1} \right)} - {r\left( t_{k} \right)}} = {\frac{1}{2}\left( {{\overset{.}{r}\left( t_{k + 1} \right)} + {\overset{.}{r}\left( t_{k} \right)}} \right)\Delta \; t_{k}}}$

where {dot over (r)}(t) stands for the velocity of the LED at time t andΔt_(k)≡t_(k+1)−t_(k). However, since a device held by hand does not haveinertia, one can ignore {dot over (r)}(t_(k)) in the above equation,simply use the following equation to compute {right arrow over (V)}:

{right arrow over (V)}={dot over (r)}(t _(k+1))*Δt _(k)  (1)

Since the pointing/input device 50 experiences both translation andspin, {dot over (r)}(t) can be expressed as

{dot over (r)}(t)=ω(t)×(r(t)−x(t))+v(t)  (2)

where ω(t) is the angular velocity of r(t) at time t, x(t) is the spinpivot of r(t) at time t, x is the cross-product notation, and v(t) isthe linear velocity of r(t) at time t. Theoretically, the value of v(t)at t_(k+1) is computed as follows:

v(t _(k+1))=v(t _(k))+a(t _(k+1))*Δt _(k)  (3)

where a(t_(k+1)) is the linear acceleration of r(t) at t_(k+1). Again,since a hand-held device does not have inertia, one can ignore the termv(t_(k)) in equation (3), simply use the following equation to computev(t_(k+1)):

v(t _(k+1))=a(t _(k+1))*Δt _(k)  (4)

Therefore, from equations (1), (2), and (4), we have the followingequation for {right arrow over (V)}:

{right arrow over (V)}=ω(t _(k+1))×(r(t _(k+1))−x(t _(k+1)))*Δt _(k)+a(t _(k+1))*(Δt _(k))²  (5)

The values of ω(t_(k+1)) and a(t_(k+1)) can be obtained respectivelyfrom the Gyroscope sensor and the G-sensor of the motion sensor module66. On the other hand, r(t_(k+1))−x(t_(k+1)) is nothing but D₂(direction of the pointing device 2 at t_(k+1), to be defined below)times L, the distance between 1104 and 1106 in FIG. 11:

r(t _(k+1))−x(t _(k+1))=L*D ₂  (6)

Hence, by substituting (6) into (5), we have the following formula for{right arrow over (V)}1108:

{right arrow over (V)}=L*(ω(t _(k+1))×D ₂)*Δt _(k) +a(t _(k+1))*(Δt_(k))²  (7)

To define D₂, let N=(x_(N), 0, z_(N)) be a vector in the direction ofnorth with unit length, i. e.,

√{square root over ((x _(N))²+(z _(N))²)}{square root over ((x _(N))²+(z_(N))²)}=1

Let the azimuth, pitch and roll angles provided by the sensors of themotion sensor module 66 of the pointing/input device 50 at t_(k+1) be α,β, and γ, respectively (see FIG. 12). The azimuth is a clockwiserotation of α about the y-axis. The rotation matrix is given by

${R_{y}(\alpha)} = \begin{bmatrix}{\cos \; \alpha} & 0 & {{- \sin}\; \alpha} \\0 & 1 & 0 \\{\sin \; \alpha} & 0 & {\cos \; \alpha}\end{bmatrix}$

The pitch is a clockwise rotation of β about the x-axis. The rotationmatrix is given by

${R_{x}(\beta)} = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos \; \beta} & {\sin \; \beta} \\0 & {{- \sin}\; \beta} & {\cos \; \beta}\end{bmatrix}$

The roll is a clockwise rotation of γ about the z-axis. The rotationmatrix is given by

${R_{z}(\gamma)} = \begin{bmatrix}{\cos \; \gamma} & {\sin \; \gamma} & 0 \\{{- \sin}\; \gamma} & {\cos \; \gamma} & 0 \\0 & 0 & 1\end{bmatrix}$

By multiplying these three matrices up, we get a single rotation matrix

$\begin{matrix}{{R\left( {\gamma,\beta,\alpha} \right)} = {{{R_{z}(\gamma)}{R_{x}(\beta)}{R_{y}(\alpha)}} = \begin{bmatrix}{{\cos \; \alpha \; \cos \; \gamma} + {\sin \; \alpha \; \sin \; \beta \; \sin \; \gamma}} & {\cos \; \beta \; \sin \; \gamma} & {{{- \sin}\; \alpha \; \cos \; \gamma} + {\cos \; \alpha \; \sin \; \beta \; \sin \; \gamma}} \\{{{- \cos}\; \alpha \; \sin \; \gamma} + {\sin \; \alpha \; \sin \; \beta \; \cos \; \gamma}} & {\cos \; \beta \; \cos \; \gamma} & {{\sin \; \alpha \; \sin \; \gamma} + {\cos \; \alpha \; \sin \; \beta \; \cos \; \gamma}} \\{\sin \; \alpha \; \cos \; \beta} & {{- \sin}\; \beta} & {\cos \; \alpha \; \cos \; \beta}\end{bmatrix}}} & (8)\end{matrix}$

To find the direction D₂ of the pointing/input device 50 at t_(k+1),simply multiply the unit vector N by the rotation matrix R(γ,β,α):

D ₂ =R(γ,β,α)N  (9)

In Step 84, the computing device 56 processor computes the intersectionpoint 70 of the axis of the pointing/input device 50 with the imagedisplay apparatus 52 at time t_(k+1) (see FIG. 9). The location of theintersection point 70 is where the operator 0 wants a cursor to be shownon the image display apparatus 52. The axis of the pointing/input device50, denoted 1118, can be expressed as

L(u)=B+t*D ₂ tεR  (10)

where B 1104 is the location of the LED light source 62 and D₂ is thepointed direction of the pointing/input device 50 at t_(k+1) (see FIG.11). B is available from Step 88 and D₂ is available from Step 90 (seeFIG. 8). Hence, to find the point 70, simply find the intersection pointof L(t) defined in equation (10) with the plane z=0.

In Step 86, the computing device 56 processor sends instructions to showa cursor at the point computed in Step 84 or move the cursor from itsprevious location to the new location computed in Step 84. The computingdevice 56 processor will also perform required actions to accommodateinteractions specified by the operator 0 through keys or buttons of thecontrol panel 64 of the pointing/input device 50.

Of course, the computing device 56 repeats the above steps for each newimage sent by the image capture device.

The foregoing discussion describes a system comprising a single imagingdevice 54, but multiple imaging devices may be incorporated into thepresently disclosed system. In one embodiment as shown in FIG. 13, apointing system is provided including two imaging devices 1302 and 1304,installed at the upper left corner and the upper right corner of thedisplay surface 52, respectively. The computation process for thecomputing device 56 processor in this case is shown schematically inFIG. 14, and is explained in detail in the discussion that follows.

In Step 1402, the work is similar to Step 80 in FIG. 8 except the worknow has to be performed for both imaging devices 1302, 1304. For theimages taken by the imaging devices 1302, 1304 at time t_(k), thecomputing device 56 processor identifies (in captured image frames) theregions corresponding to the light source 62 in these images. Thecenters of these regions will be denoted 1502 and 1504, respectively(see FIG. 15). Centers 1502 and 1504 are then mapped to points 1506 and1508 on the virtual image plane 1510 through the focal points 1512 and1514, respectively (see FIG. 15).

In Step 1404 (FIG. 14), a standard triangulation method is used to findthe location of the light source 62 in 3D space. The triangulationmethod will find the intersection point of the ray through 1502 and 1512(or through 1512 and 1506) and the ray through 1504 and 1514 (or,through 1514 and 1508). This intersection point is the location of thelight source 62 in 3D space. A suitable method for triangulation isprovided in Hartley et al., 1997, “Triangulation,” Computer Vision andImage Understanding, Vol. 68, No. 2, pp 146-157, although othertriangulation methods may be suitable and are contemplated for useherein.

Once the location of the light source in 3D space is known, theremaining steps depicted in FIG. 14 are the same as the correspondingsteps in FIG. 8, i.e., Step 1406 will use the information provided byStep 1408 to compute the direction D₂ of the pointing device 50 definedin eq. (9), and then use the 3D location of the light source and thedirection of the pointing device 50 to find an intersection point of theaxis of the pointing device 50 with the display surface 52 of the imagedisplay apparatus at time t_(k). This intersection point is the locationof the aimed point of the pointing device 50.

Summarizing, the present disclosure provides robust methods and systemsfor three-dimensional pointing using light tracking and relativeposition detection techniques. Advantageously, the disclosed methods andsystems are likewise economical, simple, and likely already available inmany homes but for the pointing/input device 50 and software. But forthe pointing/input device 50 and software, for additional hardware thesystem 10 requires only a computing device 56 and conventional imagingdevices 54 such as standard webcams, and has no requirement for anyspecific wired or wireless connection (such as wiring or cabling, or aspecialized IR or other signal) between the pointing/input device 50 andthe imaging device 54. Exemplary advantages of the disclosed systeminclude allowing an operator to point and/or input gesture commands to acomputing device, a “smart” television, and the like in 3D mode.

One of ordinary skill in the art will recognize that additionalembodiments of the invention are also possible without departing fromthe teachings herein. Thus, the foregoing description is presented forpurposes of illustration and description of the various aspects of theinvention, and one of ordinary skill in the art will recognize thatadditional embodiments of the invention are possible without departingfrom the teachings herein. This detailed description, and particularlythe specific details of the exemplary embodiments, is given primarilyfor clarity of understanding, and no unnecessary limitations are to beimported, for modifications will become obvious to those skilled in theart upon reading this disclosure and may be made without departing fromthe spirit or scope of the invention. Relatively apparent modifications,of course, include combining the various features of one or more figureswith the features of one or more of other figures. All suchmodifications and variations are within the scope of the invention asdetermined by the appended claims when interpreted in accordance withthe breadth to which they are fairly, legally and equitably entitled.

What is claimed is:
 1. A computing system for direct three-dimensionalpointing and command input, comprising: at least one computing devicehaving at least one processor, at least one memory, and at least onegraphical user interface; a pointing/input device including at least onelight source and a motion sensor module providing information regardingan absolute and a relative displacement of the pointing/input device tothe at least one computing device; at least one imaging device operablylinked to the computing device processor and configured for capturing aplurality of image frames each including a view of the at least onelight source as the pointing/input device is held and/or moved in athree-dimensional space and within a field of view of the at least oneimaging device; and at least one non-transitory computer program productoperable on the computing device processor; wherein the at least onecomputer program product includes executable instructions forcalculating at least a position and/or a motion of the at least onelight source in three-dimensional space from the plurality of sequentialimage frames and from the pointing/input device absolute and relativedisplacement information, and for rendering on the graphical userinterface a visual indicator corresponding to the calculated positionand/or the motion of the at least one light source.
 2. The system ofclaim 1, including at least two imaging devices.
 3. The system of claim1, wherein the at least one computer program product includes executableinstructions for determining a position and/or a motion of the at leastone light source in three-dimensional space by the steps of: determininga current position of the at least one light source in a captured imageframe; determining a prior position of the at least one light source inat least one prior captured image frame; determining a relativedisplacement of the pointing/input device from information provided bythe motion sensor module; calculating a location of the at least onelight source in three-dimensional space from the determined relativedisplacement information and the determined at least one light sourcecurrent and prior positions; determining a pointing direction of thepointing/input device defining an axis of the pointing/input device frominformation provided by the motion sensor module; and from thecalculated three-dimensional location of the at least one visible lightsource and the determined pointing direction of the pointing/inputdevice provided by the motion sensor module, calculating an intersectionpoint of the axis of the pointing/input device and the graphical userinterface.
 4. The system of claim 3, wherein the at least one computerprogram product includes executable instructions for displaying thecalculated intersection point as a visual indicator in the graphicaluser interface.
 5. The system of claim 3, wherein the at least onecomputer program product includes executable instructions forcalculating a location of the at least one light source inthree-dimensional space from the determined current and prior positionsof the at least one light source by: identifying a first regioncorresponding to a position of the at least one light source in thecaptured image frame and a second region corresponding to a position ofthe at least one light source in the at least one prior captured imageframe; from said identified first region and second region, calculatinga first position of the at least one light source in the captured imageframe and a second position of the at least one light source in the atleast one prior captured image frame; and from said first and secondpositions, calculating a displacement vector of the pointing/inputdevice caused by translating the at least one light source between thefirst region and the second region.
 6. The system of claim 5, whereinthe at least one computer program product includes executableinstructions for, in succeeding sets of current and prior captured imageframes, iteratively repeating the steps of identifying first and secondlocations, calculating first and second positions, and calculating adisplacement vector.
 7. The system of claim 1, wherein the at least oneimaging device has a frame capture rate of at least 30 frames persecond.
 8. In a computing system environment, a method for tracking apointing/input device, comprising: holding and/or moving apointing/input device including at least one light source and a motionsensor module in a three-dimensional space disposed within a field ofview of at least one imaging device operably connected to a computingdevice having at least one processor, at least one memory, and at leastone graphical user interface; by the at least one imaging device,capturing a plurality of sequential image frames each including a viewof a position of the at least one light source within the imaging devicefield of view; from the motion sensor module, obtaining absolute andrelative displacement information of the at least one pointing/inputdevice; by the at least one processor, calculating at least a positionand/or a motion of the at least one light source in three-dimensionalspace from said position of the at least one light source in theplurality of sequential image frames and said absolute and relativedisplacement information; and rendering a visual indicator on the atleast one graphical user interface corresponding to the position and/ormotion of the at least one point light source.
 9. The method of claim 8,further including interpreting the position and/or motion of the atleast one light source as a predetermined pointing and/or input command.10. The method of claim 8, wherein the position and/or motion of the atleast one light source in three-dimensional space is calculated by:determining a current position of the at least one light source in acaptured image frame; determining a prior position of the at least onelight source in at least one prior captured image frame; determining arelative displacement of the pointing/input device from informationprovided by the motion sensor module; calculating a location of the atleast one light source in three-dimensional space from the determinedrelative displacement information and the determined at least one lightsource current and prior positions; determining a pointing direction ofthe pointing/input device defining an axis of the pointing/input devicefrom information provided by the motion sensor module; and from thecalculated three-dimensional location of the at least one light sourceand the determined pointing direction of the pointing/input deviceprovided by the motion sensor module, calculating an intersection pointof the axis of the pointing/input device and the graphical userinterface.
 11. The method of claim 10, including displaying thecalculated intersection point as a visual indicator in the graphicaluser interface.
 12. The method of claim 10, wherein the location of theat least one light source in three-dimensional space is calculated by:identifying a first region corresponding to a position of the at leastone light source in the captured image frame and a second regioncorresponding to a position of the at least one light source in the atleast one prior captured image frame; from said identified first regionand second region, calculating a first position of the at least onelight source in the captured image frame and a second position of the atleast one light source in the at least one prior captured image frame;and from said first and second positions, calculating a displacementvector of the pointing/input device caused by translating the at leastone light source between the first region and the second region.
 13. Themethod of claim 12, further including the steps of, in succeeding setsof current and prior captured image frames, iteratively repeating thesteps of identifying first and second locations, calculating first andsecond positions, and calculating a displacement vector.
 14. A computingsystem for direct three-dimensional pointing and command input,comprising: at least one computing device having at least one processor,at least one memory, and at least one graphical user interface; apointing/input device including at least one light source and a motionsensor module providing information regarding an absolute and a relativedisplacement of the pointing/input device to the at least one processor;first and second imaging devices operably linked to the computing deviceprocessor and configured for capturing a plurality of image frames eachincluding a view of the at least one light source as the pointing/inputdevice is held and/or moved in a three-dimensional space and within afield of view of the at least two imaging devices; and at least onenon-transitory computer program product operable on the computing deviceprocessor; wherein the at least one computer program product includesexecutable instructions for calculating at least a position and/or amotion of the at least one light source in three-dimensional space fromthe plurality of sequential image frames and from the pointing/inputdevice absolute and relative displacement information, and for renderingon the graphical user interface a visual indicator corresponding to thecalculated position and/or the motion of the at least one light source.15. The system of claim 14, wherein the at least one computer programproduct determines a position and/or a motion of the at least one lightsource in three-dimensional space by the steps of: determining a currentposition of the at least one light source in corresponding currentcaptured image frames captured by the first and second imaging devices;determining a prior position of the at least one light source in atleast one corresponding prior captured image frame captured by the firstand second imaging devices; from the determined current and at least oneprior position of the at least one light source in the correspondingcurrent and prior captured image frames, calculating a location inthree-dimensional space of the at least one light source bytriangulation; determining a pointing direction of the pointing/inputdevice defining an axis of the pointing/input device from informationprovided by the motion sensor module; and from the calculatedthree-dimensional location of the at least one visible light source andthe determined pointing direction of the pointing/input device providedby the motion sensor module, calculating an intersection point of theaxis of the pointing/input device and the graphical user interface. 16.The system of claim 15, wherein the at least one computer programproduct includes executable instructions for displaying the calculatedintersection point as a visual indicator in the graphical userinterface.
 17. The system of claim 15, wherein the at least one computerprogram product includes executable instructions for calculating alocation of the at least one light source in three-dimensional spacefrom the determined current and prior positions of the at least onelight source by: identifying a first region corresponding to a positionof the at least one light source in the corresponding current capturedimage frames and a second region corresponding to a position of the atleast one light source in the at least one prior corresponding capturedimage frames; from said identified first region and second region,calculating a first position of the at least one light source in thecorresponding current captured image frames and a second position of theat least one light source in the at least one prior correspondingcaptured image frames; and calculating a location of the at least onelight source in three-dimensional space by a triangulation method. 18.The system of claim 15, wherein the first and second imaging deviceseach have a frame capture rate of at least 30 frames per second.